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ABSTRACT 


This  thesis  supports  the  Army’s  mission  to  establish  a  measure  for  cognitive 
agility  in  soldiers.  It  examines  attention-allocation  patterns  as  quantified  by  eye-tracking 
data  collected  while  subjects  played  a  military-relevant  cognitive  agility  computer  game 
(Make  Goal),  to  determine  whether  certain  patterns  are  associated  with  effective 
performance.  It  also  investigates  the  effects  of  stress  on  cognitive  agility  as  measured  by 
attention  allocation. 

Methods:  Forty  military  officers  were  randomly  assigned  to  a  stress  or  control 
group.  Stress  level  was  manipulated  by  timed  turns  and  experimenter  behavior. 

Results:  Eye-tracking  data  was  analyzed  in  terms  of  regions  of  interest  on  which 
subjects  focused  their  cognitive  workload.  Results  were  analyzed  by  stress  and  control 
group  and  top  and  bottom  ten  performers.  The  stress  and  control  groups  showed  similar 
attention-allocation  patterns.  The  high  performers  attended  more  to  the  important 
information  and  made  more  optimal  selections  than  poor  performers. 

Discussion:  Results  are  discussed  in  the  context  of  the  Yerkes-Dodson  stress 
model.  Eye-tracking  data  revealed  attention-allocation  patterns  associated  with  higher 
performance.  In  order  to  better  detect  the  impact  of  stress  on  Cognitive  Agility,  an 
experiment  that  includes  a  wider  range  of  stress  levels  is  needed. 
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EXECUTIVE  SUMMARY 


The  Army  has  identified  cognitive  dominance  as  being  essential  to  maintaining 
superiority  over  adversaries  (United  States  Army  Combined  Arms  Center,  2014). 
However,  the  Army  lacks  a  standardized  means  of  identifying  and  quantifying  cognitive 
agility,  and  this  thesis  is  part  of  a  larger  effort  by  TRADOC  to  develop  just  such  a 
measure.  In  support  of  this  effort,  this  thesis  has  a  two-fold  goal.  First,  it  examines 
attention-allocation  patterns  demonstrated  by  subjects  during  an  experiment  in  order  to 
determine  if  certain  attention-allocation  patterns  are  associated  with  effective  cognitive- 
agility  performance.  Second,  this  thesis  aims  to  investigate  the  effects  of  stress  on 
cognitive  agility  as  measured  by  attention  allocation.  Eye-tracking  data  was  used  to 
measure  attention  allocation,  as  it  provides  a  non-intrusive,  objective  and  quantifiable 
means  of  gathering  information  on  subjects’  cognitive  processes,  as  well  as  the  effects  of 
stress  on  decision  making  (Sullivan,  Yang,  Day,  &  Kennedy,  2011;  Vine,  Uiga,  Lavric, 
Moore,  Tsaneva- Atanasova,  &  Wilson,  2014).  This  thesis  is  an  exploratory  study,  and  it 
is  hoped  that  any  information  gleaned  will  be  of  use  in  directing  further  study  in  this  area. 

This  thesis  examines  attention-allocation  patterns  demonstrated  by  subjects  while 
they  played  a  military-relevant  cognitive  agility  computer  game  called  Make  Goal.  Make 
Goal  requires  subjects  to  assess  and  use  given  items  of  information  about  a  recruiting 
region  (i.e.,  the  number  of  individuals  qualified  for  military  service  in  that  region). 
Subjects  must  then  decide  which  items  of  information  are  important  and  use  them  to 
make  optimal  decisions  regarding  recruiter  placement  to  gain  the  maximum  number  of 
recruits  within  one  hundred  timed  trials.  The  relative  importance  of  these  information 
items  changes  throughout  the  game,  requiring  subjects  to  detect  the  change  and  shift 
strategy  accordingly.  Attention  allocation  was  measured  by  collecting  eye-tracking  data 
as  each  subject  played  the  game.  The  game  screen  was  split  into  various  regions  of 
interest  (ROIs)  so  that  the  eye-tracking  data  could  be  analyzed  to  determine  at  which 
information  a  subject  was  directing  his  or  her  attention  during  the  game  and  for  how  long. 
Cognitive  workload  data  also  was  collected  via  measurement  of  pupil  dilation. 
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The  subjects  in  this  experiment  consisted  of  forty  military  officer  volunteers  who 
were  randomly  assigned  to  either  the  stress  or  control  group.  Those  in  the  stress  group 
were  subjected  to  a  social  stress  induction,  in  which  the  experimenter  monitored  them 
closely  while  they  played  the  game,  with  periodic  comments  regarding  their  game 
performance.  In  contrast,  those  in  the  control  group  received  no  such  monitoring  by  the 
experimenter.  It  was  expected  that  the  stress  group  would  either  do  much  better  or  much 
worse  than  the  control  group,  according  to  the  Yerkes-Dodson  inverted  U  relationship 
between  stress  and  performance.  The  Yerkes-Dodson  model  postulates  that  both  very 
high  and  very  low  levels  of  stress  can  negatively  impact  performance,  while  moderate 
stress  can  actually  improve  performance  (Yerkes  &  Dodson,  1908).  In  order  to  determine 
if  certain  attention-allocation  patterns  are  associated  with  effective  cognitive  agility, 
performance  eye -tracking  data  also  was  examined  among  the  best  and  worst  performers. 

Analysis  of  the  results  (both  the  Make  Goal  Game  results  and  the  eye-tracking 
measures)  showed  almost  no  significant  differences  in  attention-allocation  patterns  and 
performance  between  the  stress  and  control  groups.  Only  one  significant  difference  was 
found;  the  control  group  looked  more  often  at  one  particular  ROI,  Turn  Remaining/ 
Progress  (which  was  not  strategically  important  to  performance).  However,  several 
differences  were  found  between  the  high-  and  low-performing  groups.  As  might  be 
expected,  high  performers  were  better  able  to  identify  the  most  relevant  pieces  of 
information  and  use  this  information  to  make  optimal  recruiting  decisions. 

The  precise  role  of  stress  in  affecting  performance  in  this  study  was  not  clear  from 
the  analysis,  as  almost  no  significant  differences  were  found  in  attention-allocation 
patterns  between  the  stress  and  control  groups.  Subjective  levels  of  stress  were  reported 
in  a  post-task  survey.  While  the  stress  group  reported  the  experimenter’s  behavior  as 
being  more  stressful  than  the  control  group,  the  control  group  was  not  entirely  free  from 
stress;  eleven  out  of  the  twenty  control-group  subjects  reported  the  game  timer  as  being 
stressful.  The  top  ten  high  performers  and  bottom  ten  low  performers  both  shared  mixes 
of  subjects  from  both  the  stress  and  control  groups.  Interestingly,  though,  the  high 
performers  reported  the  timer  as  being  significantly  more  stressful  than  the  low 
performers  did.  This  finding  is  consistent  with  the  Yerkes-Dodson  relationship  between 


stress  and  performance,  in  which  moderate  stress  is  associated  with  improved 
performance.  In  order  to  better  analyze  the  effects  of  stress,  some  experimental  design 
improvements  could  be  made,  including  removal  of  the  timer  from  the  game  interface  for 
the  control  group  so  that  the  control  group  encounters  no  stressors.  Perhaps  an  EEG  or 
skin  conductance  test  would  be  useful,  in  order  to  provide  a  more  specific  classification 
of  stress  levels  than  that  provided  by  the  eye  tracking  and  subjects’  self-reports.  A  more 
precise  and  standardized  classification  of  stress  levels  would  provide  a  better  test  of  the 
Yerkes-Dodson  model. 

In  conclusion,  the  ultimate  goal  of  this  TRADOC  study  is  first,  to  identify 
cognitively  agile  military  personnel  and  second,  to  investigate  the  effects  of  stress  on 
cognitive  agility.  This  thesis  found  that  eye  tracking  did,  in  fact,  provide  insight  into  the 
decision-making  processes  of  more  cognitively  agile  subjects;  it  revealed  that  they  found 
and  then  focused  on  more  relevant  information  and  therefore  performed  better  in  the 
game.  Although  the  results  comparing  the  stress  and  control  groups  were  not  what  was 
expected,  the  fact  that  eye-tracking  data  proved  useful  in  identifying  attention-allocation 
patterns  between  high  and  low  performers  shows  promise  for  future  work. 
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I. 


INTRODUCTION 


A.  BACKGROUND 

In  the  Army’s  Force  2025  and  Beyond  planning  process,  cognitive  dominance 
was  identified  as  an  essential  component  to  maintaining  a  “decisive  edge”  over 
adversaries  (United  States  Army  Combined  Arms  Center,  2014).  In  support  of  this 
emphasis  on  cognitive  agility,  the  TRADOC  Analysis  Center  has  undertaken  a  study 
seeking  to  (1)  operationalize  and  measure  cognitive  agility  in  a  military  context  and  (2) 
determine  if  psychological  measures  of  cognitive  flexibility,  focused  attention  and 
creativity  correlate  with  outcome  measures  of  a  military  cognitive-agility  computer  game. 
Although  aspects  of  cognitive  agility  have  been  studied  in  the  civilian  sector  (Good, 
2014),  most  of  these  studies  lack  the  stress  and  high  stakes  of  a  military  environment. 
Therefore,  the  TRADOC  study,  which  is  the  focus  of  this  thesis,  aims  to  create  a  military¬ 
relevant  measure  of  cognitive  agility  that  replicates  some  of  the  stressors  typically 
experienced  by  military  decision  makers.  The  ultimate  goal  of  this  body  of  work  is  first, 
to  identify  cognitively  agile  military  personnel  and  second,  to  investigate  the  effects  of 
stress  on  cognitive  agility.  Thus,  it  is  important  to  understand  the  attention-allocation 
patterns  used  by  the  cognitively  agile,  because  these  methods  can  uncover  underlying 
cognitive  strategies  (Sullivan,  Yang,  Day,  &  Kennedy,  2011).  This  thesis  supports  the 
Army’s  goal  by  examining  attention-allocation  patterns  as  they  are  quantified  by  eye¬ 
tracking  data,  seeking  to  determine  if  certain  attention-allocation  patterns  are  associated 
with  effective  cognitive-agility  performance,  and  by  investigating  the  effects  of  stress  on 
cognitive  agility  as  indicated  by  attention  allocation. 

The  means  used  in  this  study  to  gather  information  about  attention  allocation  and 
underlying  strategy  was  eye  tracking.  Eye-tracking  data  is  objective  and  quantifiable  and 
can  provide  insight  into  such  cognitive  processes  as  decision  making  (Spivey,  Richardson 
&  Fitneva,  2004).  This  thesis  focuses  on  a  military  relevant  cognitive-agility  computer 
game  called  “Make  Goal,”  in  which  subjects  have  to  allocate  recruiters  to  regions  in 
order  to  meet  recruiting  goals  within  a  certain  number  of  trials  (Alt,  Appleget,  Buttrey, 
Kennedy,  Sciarini,  &  Johnson,  2015).  In  order  to  make  optimal  recruiting  decisions, 
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subjects  must  rely  on  particular  pieces  of  information  such  as  the  map  of  the  recruiting 
districts,  amount  of  time  remaining,  overall  goal,  number  of  personnel,  and  output. 
Periodically,  the  relative  importance  of  each  piece  of  information  changes,  resulting  in  a 
need  to  shift  one’s  strategy.  Thus,  the  game  measures  how  well  subjects  attend  to 
feedback  and  how  quickly  they  realize  that  they  need  to  change  strategy. 

Subjects  played  Make  Goal  under  one  of  two  conditions.  A  stress  group  had  the 
experimenter  closely  monitoring  them  while  they  played  the  game  (experimenter-induced 
stress),  and  the  control  group  was  not  subject  to  this  stress.  Eye-tracking  data  was 
collected  on  each  subject  during  the  game.  During  the  original  study  of  the  Make  Goal 
game,  the  eye  tracking  results  indicated  the  ability  to  discern  subjects’  transition  from 
exploration  to  exploitation  of  the  game  (Alt  et  al.,  2015).  It  is  hoped  that  similar  insights 
can  be  gleaned  from  this  study  by  comparing  gaming  performance  of  the  subjects  and 
their  eye-tracking  data,  to  establish  whether  patterns  emerge  indicating  that  more 
successful  subjects  have  different  eye-tracking  patterns  from  the  less  successful. 

The  impact  of  stress  on  cognitive  performance  is  well  documented  (Eysenck, 
Derakshan,  Santos,  &  Calvo,  2007),  and  the  measurability  of  this  impact  by  using  eye¬ 
tracking  data  has  been  established.  According  to  Wilson  (2012)  as  cited  in  Vine,  Uiga, 
Lavric,  Moore,  Tsaneva-Atanasova  and  Wilson  (2014),  anxiety  is  known  to  cause 
disruption  in  a  focused  gaze  leading  to  “more,  shorter  fixations  to  a  variety  of  locations, 
and  (subjects)  are  unable  to  maintain  the  long,  target-focused  fixations  important  for  the 
planning  and  control  of  movement”  (p.  469).  Thus,  previous  work  suggests  that  stress 
should  influence  attention  allocation  during  the  Make  Goal  Game. 

B.  LITERATURE  REVIEW 

1.  Eye  Tracking 

As  mentioned  above,  the  use  of  eye  tracking  in  order  to  quantify  cognitive 
processes  has  been  established  (Poole,  Ball  &  Phillips,  2005;  Spivey  et  al.,  2004;  Sullivan 
et  al.  2011).  However,  the  majority  of  the  relevant  studies  using  eye-tracking  metrics  as  a 
means  of  quantifying  cognitive  processes  have  involved  pilots  (Bellenkes,  Wickens  & 
Kramer,  1997;  Kasarskis,  Stehwien,  Hickox,  Aretz,  &  Wickens,  2001;  Sarter,  Mumaw  & 
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Wickens,  2007;  Sullivan  et  al.,  2011;  Vine  et  al.,  2014)  or  drivers  (Konstantopoulos, 
Chapman  &  Crandall,  2010;  Maltz  &  Shinar,  1999;  McPhee,  Scialfa,  Dennis,  Ho,  & 
Caird,  2004;  Wikman  &  Summala,  2005).  These  studies  involve  subjects  performing  a 
task  with  which  they  are  familiar  and  subjects  who  can  be  categorized  on  a  spectrum 
from  novice  to  experienced.  Therefore,  the  eye -tracking  data  has  been  used  to  compare 
the  differences  in  the  decision-making  processes  of  subjects  across  this  spectrum  of 
varied  experience. 

The  use  of  eye  tracking  in  this  thesis  differs  fundamentally  from  these  other 
studies.  Rather  than  trying  to  understand  how  subjects’  cognitive  processes  vary  with  or 
relate  to  their  experience  level,  it  seeks  to  understand,  or  at  least  quantify,  the  cognitive 
process  itself,  according  to  how  subjects  performed  on  a  decision-making  task  with 
which  they  had  no  prior  experience.  Studies  with  pilots  have  found  that  better  decisions 
are  made  by  the  more  experienced  pilots  due  to  the  fact  that  they  are  more  efficient  and 
effective  in  how  they  allocate  their  attention  (Schriver,  Morrow,  Wickens  &  Talleur, 
2008;  Sullivan  etal.,2011). 

This  thesis  investigated  whether  attention-allocation  patterns  can  identify  the 
more  cognitively  agile.  It  examined  the  eye-tracking  data  of  subjects  as  they  played  Make 
Goal,  which  required  them  to  shift  strategies  throughout  the  game’s  duration.  The  thesis 
examined  whether  attention-allocation  patterns  differed  between  high  and  low  performers 
and  between  the  stress  or  control  group.  With  regard  to  the  eye  tracking,  this  study 
focuses  on  fixation-duration  data  as  well  as  workload  (indicated  by  pupil  dilation).  These 
measures  are  typically  used  in  correlating  eye  tracking  and  cognitive  processes.  The  three 
eye-tracking  measures  used  most  frequently  in  cognitive  analysis  are  saccades,  fixations 
and  pupil  dilation.  Although  this  study  does  not  focus  on  saccades,  per  se,  it  is  worth 
noting  what  a  saccade  is  because  fixations,  which  this  study  does  examine,  are  defined  by 
saccades.  A  saccade  is  defined  as  the  quick  movement  of  the  gaze  lasting  between  30  and 
50  milliseconds  going  from  consecutive  points  of  interest,  while  a  fixation  is  defined  as 
the  time  between  saccades,  usually  lasting  200-300  milliseconds  (McCarley  &  Kramer 
2007).  Fixations,  which  are  one  of  the  primary  areas  of  interest  in  this  study,  are 
particularly  important  as  it  is  during  fixations  that  a  subject  acquires  information 
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(McCarley  &  Kramer  2007).  “A  number  of  studies  have  shown  that  gaze  duration  is 
sensitive  to  the  information  value  of  the  item  being  fixated,  with  items  that  are 
unexpected  or  semantically  inconsistent  with  their  context  receiving  longer  gazes  than 
items  that  are  expected  within  the  context  (e.g.,  Friedman,  1979;  Loftus  &  Mackworth, 
1978),”  as  cited  by  McCarley  and  Kramer  (2007,  p.  97).  Furthermore,  “a  number  of 
studies  have  shown  that  gaze  duration  (what  we  in  this  study  are  calling  a  fixation)  is 
sensitive  to  the  information  value  of  the  item  being  fixated”  (McCarley  &  Kramer  2007, 
p.  97).  The  link  between  pupil  dilation  and  cognitive  processes  is  established;  generally,  a 
subject’s  pupils  will  dilate  in  accord  with  a  higher  cognitive  workload  (Kahneman,  1973; 
Marshall,  2007). 

The  study  focuses  on  subjects’  visual  fixations  on  regions  of  interest  (ROI)  within 
the  Make  Goal  Task.  In  the  context  of  the  study,  an  ROI  is  simply  an  area  on  the 
computer  screen  that  provides  meaningful  information  to  the  subject.  The  examination  of 
the  breakdown  of  subjects’  fixations  on  different  ROIs  on  the  screen  can  yield 
information  about  what  they  considered  important,  or  what  given  information  they  were 
using  to  inform  their  decisions.  Finally,  the  link  between  pupil  dilation  and  cognitive 
processes  is  established  (Marshall,  2007).  Specifically,  a  subject’s  pupils  will  dilate  when 
they  experience  an  increase  in  cognitive  workload.  (Kahneman,  1973).  Thus,  larger 
dilation  is  positively  associated  with  higher  cognitive  workload. 

2.  Stress 

Stress  is  an  important  aspect  of  this  study  because  it  is  an  intrinsic  part  of  the 
military  environment;  understanding  its  effects  on  the  cognitive  agility  of  decision 
makers  is  one  of  the  Army’s  key  objectives  in  this  study.  There  are  varying  definitions  of 
stress;  the  one  used  for  this  thesis  is  as  defined  by  Stokes  and  Kite  (2001).  Stress  is  “the 
result  of  a  mismatch  between  individuals’  perceptions  of  the  demands  of  the  task  or 
situation  and  their  perceptions  of  the  resources  for  coping  with  them”  (Stokes  &  Kite, 
2001,  p.  116).  Just  as  there  are  varying  definitions  of  stress  itself,  the  precise  impact  of 
stress  on  cognitive  performance  is  also  the  subject  of  study  and  debate.  It  seems  to  be  the 
finding  of  the  majority  of  studies  that  stress  adversely  impacts  cognitive  performance, 
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from  memory  to  decision  making  to  attention  and  perceptual-motor  performance 
(Ashcraft  &  Kirk,  2001;  Ashcraft,  2002;  Braunstein-Bercovitz,  Dimentman- Ashkenazi  & 
Lubow,  2001;  Enander  1989;  Eysenck  et  al.,  2007;  Friedland,  Keinan  &  Regev,  1992; 
Lieberman,  Bathalon,  Falco,  Georgelis,  Morgan,  Niro  &  Tharion,  2002;  Matthews  & 
Desmond,  2002;  Ozel,  2001;  Robert  &  Hockey  1997;  Van  Galen  &  van  Huygevoort, 
2000). 

There  are  other  studies  that  seem  to  indicate  that  some  types  of  stress  may 
actually  improve  cognitive  performance.  For  example,  a  study  conducted  by  Favine, 
Sibert,  Gokturk,  and  Dickens  (2002),  wherein  subjects  were  exposed  to  stress  in  the  form 
of  random  noise  bursts,  found  their  performance  on  a  vigilance  task  improved.  Similar 
findings  were  obtained  in  a  study  by  Kirk  and  Hecht  (1963),  also  involving  noise  stress. 
Another  study  conducted  by  Ozel  (2001)  involved  fire-fighters’  decision-making 
processes  in  egress  from  a  fire.  Ozel  (2001)  found  that  when  exposed  to  stress  in  the  form 
of  time  and  fire,  low  amounts  of  stress  resulted  in  better  decisions  regarding  when  to  exit, 
whereas  higher  stress  levels  resulted  in  degraded  decision  making.  As  cited  by  Staal 
(2004),  Chajut  and  Algom  (2003)  found  that  in  some  cases  “stress  reduced  interference 
and  improved  selective  attention”  (Staal,  2004,  p.  42).  Braunstein-Bercovitz’ s  study 
(2003),  cited  by  Staal  (2004)  “demonstrates  that  there  is  an  interaction  effect  between 
workload  and  psychological  stress.  When  workload  is  relatively  low  and  stress  is  high, 
the  selective  attention  effect  is  present  (negative  priming  is  attenuated).  On  the  other 
hand,  when  both  workload  and  stress  are  high,  support  for  the  selective  attention 
hypothesis  diminishes  and  the  negative  priming  effect  is  strong”  (Staal,  2004,  p.  41). 
Negative  priming  refers  to  the  phenomenon  wherein  after  making  a  conscious  effort  to 
ignore  a  stimulus,  subsequent  attempts  to  process  that  stimulus  will  be  impaired  (Tipper, 
1985).  For  example,  if  a  subject  is  told  to  focus  on  picking  a  green  pen  out  of  a  bunch  of 
blue  pens,  and  afterwards  is  told  to  focus  on  picking  out  a  blue  pen,  the  subject’s  initial 
difficulty  in  switching  from  focusing  on  the  green  pen  to  the  blue  pen  would  be  an 
example  of  negative  priming. 

The  above  findings  suggest  that  stress  can  be  beneficial  or  detrimental  to 
cognitive  processes,  depending  on  the  type  and  level  of  stress,  the  task  being  measured 
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and  also  the  individual  subject,  e.g.,  subjects  already  nervous  or  anxious  by  nature  tend  to 
show  degraded  cognitive  performance  under  stress  (Wofford,  Goodwin  &  Daly,  1999; 
Wofford,  2001;  Wofford  &  Goodwin,  2002).  This  non-monotonic  relationship  between 
stress  and  performance  was  first  proposed  by  Yerkes  and  Dodson  (1908).  This  model 
proposes  that  the  relationship  between  stress  and  performance  can  be  modeled  by  an 
inverted  U,  wherein  very  low  and  very  high  levels  of  stress  are  detrimental  to 
performance,  but  a  moderate  level  of  stress  corresponds  to  better  performance. 
Quantifying  the  effects  of  stress  on  an  individual  subject  still  presents  a  challenge 
because  the  same  stimuli  can  be  perceived  as  stressful  by  some  individuals  and  not 
stressful  by  others,  and  even  if  a  stimulus  is  perceived  as  stressful,  individual  reactions 
can  still  differ  (Cerin,  Szabo,  Hunt,  &  Williams,  2000).  Thus,  eye  tracking  provides  one 
method  for  quantifying  individual  differences  in  stress  response  across  subjects.  The 
measurability  of  this  impact  by  using  eye-tracking  data  has  likewise  been  established. 
According  to  Wilson  (2012)  as  cited  in  Vine  et  al.  (2014),  anxiety  is  known  to  cause 
disruption  in  a  focused  gaze  leading  to  “more,  shorter  fixations  to  a  variety  of  locations, 
and  (subjects)  are  unable  to  maintain  the  long,  target-focused  fixations  important  for  the 
planning  and  control  of  movement”  (Vine  et  al.,  2014,  p.  469).  Studies  conducted  on 
surgeons,  athletes,  and  pilots  have  demonstrated  that  disruptions  in  visual  fixations  have 
been  linked  to  degraded  performance  (Vine  et  al.,  2014).  For  example,  among  pilots  who 
completed  either  a  stressful  or  non-stressful  scenario,  the  pilots  in  the  stressful  scenario 
showed  more  disruptions  in  visual  fixations,  “higher  search  rates  and  increased 
randomness  in  scanning  behavior”  (Vine  et  al.,  2014,  p.  12).  Thus,  previous  work 
suggests  that  stress  should  influence  attention  allocation  during  the  Make  Goal  Game. 
Specifically,  we  would  expect  to  see  that  subjects  in  the  stress  condition  will  exhibit 
more,  shorter  fixations  and  greater  workload  as  indicated  by  pupil  size. 

3.  Literature  Review  Summary 

The  current  literature  provides  a  background  and  starting  point  for  this  study.  It 
indicates  that  stress  can  be  a  subjective  influence,  depending  upon  the  individual,  and  that 
it  is  likely  to  have  some  effect,  either  positive  or  negative,  upon  the  subject’s  cognitive 

processes.  The  social  induction  stress  used  in  this  thesis  would  most  likely  correspond  to 
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a  moderate  level  of  stress.  Based  on  the  literature,  and  particularly,  the  Yerkes-Dodson 
model  of  stress  and  performance,  improved  performance  might  be  expected  from  the 
stress  group.  Furthermore,  eye  tracking  should  be  a  reliable  means  of  quantifying  the 
effects  of  stress  upon  subjects’  cognitive  processes.  We  expect  to  see  differences  in  the 
pupil  dilation  and  fixations  exhibited  by  the  two  groups  in  this  study. 

C.  OVERVIEW  OF  THESIS  EXPERIMENT  AND  RESEARCH  QUESTIONS 

As  previously  mentioned,  this  thesis  explores  part  of  a  study  conducted  by  TRAC 
Monterey  in  order  to  address  U.S.  Army  objectives  regarding  identifying  and  quantifying 
cognitive  agility  in  its  personnel.  The  study  involved  several  traditional  psychological 
tests  of  cognitive  agility,  including  the  Stroop  Test  (Stroop,  1935),  the  Go  No  Go  Test 
(Donders,  1969),  and  the  Alternate  Use  Test  (Guilford,  Christensen,  Merrifield  &  Wilson, 
1978),  with  the  addition  of  the  Make  Goal  recruiting  scenario  game,  during  which 
subjects  were  part  of  either  a  control  or  stress  group,  and  eye -tracking  data  was  collected. 
Each  of  these  components  will  be  addressed  further  in  the  methods  chapter,  but  this  thesis 
focuses  specifically  on  the  Make  Goal  task  and  analysis  of  the  accompanying  eye¬ 
tracking  data. 

This  is  a  preliminary  study  conducting  exploratory  analysis,  and  as  such  does  not 
have  any  formal  hypotheses;  however,  the  analysis  has  the  following  objectives: 

1.  To  determine  whether  there  is  an  association  between  attention-allocation 
patterns  and  performance  on  the  Make  Goal  computer  game. 

2.  To  determine  whether  social  stress  affects  attention  allocation  patterns. 


7 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


8 


II.  METHODOLOGY 


A.  PARTICIPANTS 

The  subjects  in  this  study  consisted  of  forty  military  officers,  both  U.S.  and 
international,  from  all  service  branches.  See  Table  1  for  demographic  characteristics  of 
the  sample.  Subjects  were  all  volunteers  recruited  from  among  the  students  and  staff  at 
the  Naval  Postgraduate  School.  Recruitment  was  conducted  by  word  of  mouth,  mass 
emails,  and  the  posting  of  an  advertisement  on  the  school’s  daily  muster  page,  where 
students  are  required  to  check  a  list  of  daily  announcements.  As  subjects  responded  to 
participate,  they  were  randomly  assigned  to  one  of  two  groups:  a  stress  group  or  a  control 
group.  Statistical  analysis  demonstrated  that  there  are  no  statistically  significant 
differences  between  the  stress  and  control  groups  in  terms  of  service,  rank,  years  in 
service,  age  or  gender. 


Table  1.  Subject  Demographic  Data 


Demographic  Data 

Control  Group 

Stress  Group 

Service 

US  Army:  4 

US  Army:  4 

US  Marine  Corps:  6 

US  Marine  Corps:  3 

US  Navy:  10 

US  Navy:  12 

International:  1 

Rank 

05:  1 

04:  9 

04:7 

03:9 

03:9 

01:  1 

02:4 

Years  in  Service 

Mean  (sd) 

11.05  (6.14) 

9  (4.76) 

Age 

Mean  (sd) 

33.7  (6.28) 

30.7  (4.49) 

Gender 

14  male 

10  male 

6  female 

10  female 
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B.  MEASURES 


1.  Make  Goal  Task 

This  task  is  a  recruiting  scenario  game  in  which  subjects  are  given  one  hundred 
timed  turns  in  which  to  place  recruiters  in  various  districts  to  generate  enough  recruits  in 
order  to  meet  a  given  goal  of  seven  hundred  twenty-five  recruits.  The  time  in  which  to 
make  a  move  on  each  turn  becomes  progressively  shorter  over  the  course  of  the  game: 
subjects  begin  with  one  hundred  seconds  per  turn  and  end  with  only  fifteen  seconds  per 
turn.  Certain  demographic  data  (labeled  in  Figure  1  as  regional  data  categories)  are  given 
in  a  column  on  the  left-hand  part  of  the  screen;  subjects  can  use  this  demographic  data  to 
try  and  ascertain  the  best  possible  district  in  which  to  place  their  recruiter  on  each  turn. 
The  demographic  data  include  the  number  of  youths  between  the  ages  of  seventeen  and 
twenty-four,  the  number  of  individuals  qualified  for  military  service  and  the  number  of 
members  of  minority  groups.  Subjects  can  click  on  any  of  these  attributes  to  see  the 
districts  on  the  map  highlighted  according  to  these  attributes  (i.e.,  the  region  with  the 
most  youth  between  the  ages  of  seventeen  and  twenty-four  would  be  dark  green,  the 
region  with  second  most  would  be  light  green,  etc.)  On  the  right  side  of  the  screen  the 
names  of  each  of  the  recruiting  districts  are  listed  (labeled  in  Figure  1  as  “station 
information  based  on  regional  data”).  During  the  game  the  regions  are  not  labeled  on  the 
map  itself;  subjects  can  click  on  station  information  to  view  each  district  highlighted  on 
the  map.  Subjects  must  ascertain  how  each  of  these  attributes  contributes  to  the  number 
of  recruits  generated  in  order  to  make  optimal  recruiter  placement  decisions.  After 
submitting  each  turn  by  clicking  on  the  next  turn  button,  subjects  receive  immediate 
feedback  regarding  the  number  of  recruits  generated  on  the  last  turn  as  well  as  by  how 
much  they  missed  their  goal,  if  at  all.  The  relative  importance  of  each  piece  of 
information  (which  influences  the  “optimal  selection”)  changes  six  times  over  the  course 
of  the  game,  resulting  in  a  need  for  the  subject  to  detect  the  change  and  accordingly  shift 
strategy: 

•  Turns  1-20:  Focusing  on  the  data  attribute  diversity  (number  of  members 
of  minority  groups)  will  lead  to  the  optimal  number  of  recruits. 
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•  Turns  21-38:  The  most  salient  data  attribute  for  these  turns  is  population 
qualified  for  military  service. 

•  Turns  39-55:  The  most  salient  data  attribute  for  these  turns  is  diversity 
(number  of  members  of  minority  groups). 

•  Turns  56-71:  The  most  salient  data  attribute  for  these  turns  is  the  number 
of  youth  between  17  and  24. 

•  Turns  72-87:  The  most  salient  data  attribute  for  these  turns  is  the 
population  qualified  for  military  service. 

•  Turns  88-100:  The  most  salient  data  attribute  for  these  turns  is  diversity 
(number  of  members  of  minority  groups). 

(Appendix  A  shows  a  table  listing  the  optimal  station  selections  and  how  these  changed 
throughout  the  game).  As  shown  in  Figure  2,  subjects  also  receive  information  regarding 
the  amount  of  time  left  in  that  turn,  the  total  number  of  recruits  (accessions)  they  have 
recruited  so  far,  and  the  number  of  turns  remaining.  For  the  purposes  of  this  thesis,  the 
main  outcome  measure  was  total  recruits.  Figure  1  depicts  the  Make  Goal  game  screen 
and  Table  2  lists  the  relevant  Make  Goal  variables. 
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Figure  1.  Make  Goal  Screen 


Table  2.  Make  Goal  Game  Terminology  Definitions 


Make  Goal  Measure 

Definition 

Total  Accessions 

Total  number  of  recruits  gained  during  the 
Make  Goal  game 

Regret 

Total  number  of  possible  recruits  missed 
during  the  Make  Goal  game 

%  Optimal  Selection 

Once  a  subject  makes  an  optimal  selection 
(see  Chapter  HI,  section  B,  part  1  for 
optimal  selection  definition)  the 
percentage  of  trials  for  which  the  subject 
continues  to  stay  with  that  selection  while 
it  remains  optimal 

Time 

Duration  of  time  taken  to  complete  the 
Make  Goal  game 
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2.  Stress  Manipulation 

There  were  two  sources  of  stress  manipulated  in  this  experiment:  experimenter- 
induced  stress  and  timer  stress. 

a.  Experimenter-Induced  Stress 

The  randomly  assigned  members  of  the  stress  group  played  Make  Goal  with  the 
experimenter  observing  them  closely  and  making  comments  as  they  played  the  game.  A 
script  was  followed  by  the  experimenter  to  ensure  consistency  for  all  subjects.  At  the 
following  intervals  the  following  comments  were  made: 

•  Move  35:  “Are  you  sure  you  want  to  make  that  move?” 

•  Move  50:  “You  might  want  to  take  a  look  at  the  timer.” 

•  Move  70:  “Usually  people  are  doing  a  little  better  by  this  point.” 

•  Move  85:  “Might  want  to  check  on  that  timer  again.” 

•  Move  90:  “It  doesn’t  look  like  you’re  doing  too  well.” 

The  randomly  assigned  members  of  the  control  group  played  Make  Goal  without  the 
experimenter  observing  them  closely  or  making  any  comments  on  their  performance. 

b.  Timer  Stress 

The  Make  Goal  screen  has  a  timer  in  the  upper  right-hand  comer  of  the  screen 
(see  Figure  1).  The  time  in  which  to  make  a  move  on  each  turn  becomes  progressively 
shorter  over  the  course  of  the  game:  subjects  begin  with  ninety  seconds  per  turn  and  end 
with  only  fifteen  seconds  per  turn.  Due  to  constraints  in  game  design,  all  subjects,  both  in 
the  control  and  stress  groups,  experienced  timer  stress. 

3.  Typical  Psychological  Measures  of  Cognitive  Agility 

a.  Stroop  Test  (Stroop,  1935) 

In  the  Stroop  Test,  subjects  are  shown  a  series  of  names  of  colors  written  in 

different  color  fonts.  They  have  to  react  as  quickly  as  possible  to  the  color  in  which  the 
word  is  written  while  ignoring  what  the  word  itself  reads,  by  hitting  color  coded  arrow 
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keys  on  the  keyboard.  For  example,  in  Figure  2,  the  correct  responses  would  be  red  and 
green,  respectively.  The  test  is  scored  by  calculating  the  percentage  of  correct  responses. 


Stroop  Test  Example 


Figure  2.  Stroop  Test  Screenshot 
b.  Go  No  Go  Test  (Donders,  1969) 

This  test  is  an  eight-minute -long  test  of  focused  attention  and  reaction  time, 
wherein  subjects  are  shown  a  series  of  shapes  that  were  either  green  or  blue.  If  the  shape 
is  green,  they  have  to  hit  the  space  bar  as  quickly  as  possible;  otherwise,  they  are  not 
supposed  to  do  anything  (see  Figure  3).  Thus,  over  time,  the  subject  must  refrain  from  the 
automatic  response  of  hitting  the  space  bar  when  they  are  presented  with  a  blue  shape. 
The  test  is  scored  by  calculating  the  percentage  of  correct  responses. 
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Possible  Go  No  Go  Test 
Examples 


Screen  1  Screen  2  Screen  3 


Figure  3.  Go  No  Go  Test  Screenshot 

c.  Alternate  Use  Test  (AUT)  (Guilford,  Christensen,  Merrifield  &  Wilson, 

1978) 

The  AUT  is  a  creativity  test  in  which  subjects  are  given  three  minutes  to  list  as 
many  unique  uses  as  they  can  imagine  for  a  brick.  In  this  study,  subjects  typed  in  their 
responses  on  the  computer.  The  score  is  the  total  number  of  non-repeated  responses 
listed. 

4.  Surveys 

a.  Demographic  Survey 

This  survey  collected  information  on  subjects’  service  branch,  age,  time  in 
service,  recruiting  experience  (if  any,  and  how  long)  and  time  spent  playing  computer 
games  each  week  (if  at  all,  which  type). 

b.  Post  Make  Goal  Accessions  Survey 

Subjects  were  asked  to  list  everything  they  believed  influenced  the  number  of 
accessions  they  achieved  each  turn. 
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c.  Post  Task  Survey 

In  this  survey,  subjects  answered  questions  regarding  their  strategy  during  the 
Make  Goal  Task,  specified  which  pieces  of  information  they  found  most  useful,  and  rated 
how  stressful  they  found  the  game  and  various  elements  of  the  game  on  a  1-7  Likert 
scale,  including  the  timer  and  experimenter’s  behavior.  See  Appendix  B  for  a  copy  of  the 
survey. 


5.  Eye  Tracking 

Table  3  lists  the  eye-tracking  terminology  used  in  this  thesis  along  with  brief 
definitions.  These  measures  are  used  to  quantify  subjects’  attention  allocation  while  they 
played  Make  Goal. 


Table  3.  Eye -Tracking  Terminology  Definitions 


Eye -Tracking  Measure 

Definition 

Fixation 

Maintaining  visual  gaze  on  a  single 
location,  typically  lasting  200-300  ms. 

Number  of  Fixations 

How  many  times  each  subject  fixated 
during  the  experiment. 

Fixation  Duration  Overall 

Overall  average  length  (in  seconds)  of  each 
subject’s  fixations  during  the  experiment. 

Fixation  Duration  by  ROI 

Overall  average  length  (in  seconds)  of  each 
subject’s  fixations  in  each  ROI. 

%  Fixation  Time  by  ROI 

Time  a  subject  spent  fixated  in  each  ROI  as 
a  percentage  of  total  fixation  time 
throughout  the  game  duration. 

Number  of  Fixation  Shifts 

How  many  times  a  subject’s  fixation 
shifted  from  one  ROI  to  another  during  the 
experiment. 

Fixation  Shift  Rate 

How  many  times  a  subject’s  fixation 
shifted  from  one  ROI  to  another  during  the 
experiment  divided  by  the  total  number  of 
fixations. 

Workload 

Measure  of  subject’s  effort  according  to 
pupil  dilation,  on  a  scale  from  .0001  to 

1 .0000  in  which  higher  values  indicate 
higher  cognitive  workload. 
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During  the  Make  Goal  Task,  subjects’  eye  movements  were  measured  and 
recorded  using  the  FaceLAB  eye-tracking  equipment  and  Eye  Works  software.  Eye¬ 
tracking  data  is  collected  60  times  per  second;  pupil  dilation  data  is  collected  once  per 
second.  Eye  movements  are  measured  in  gaze  duration  in  each  region  of  interest  (ROI), 
percent  of  time  subjects  looked  at  each  ROI,  number  of  gaze  shifts  between  each  ROI, 
and  workload  as  measured  by  pupil  dilation  throughout  the  task.  There  were  six  ROIs: 
the  timer  and  the  list  of  recruiting  stations,  the  NRC  data,  the  regret  number  (number  of 
recruits  by  which  goal  was  missed),  the  map,  the  number  recruited  on  the  last  turn,  the 
number  of  turns  remaining  as  well  as  the  progress  being  made  towards  making  goal.  (See 
Figure  4  for  the  ROI’s  as  well  as  sample  eye  tracking  data  overlay.) 


TURNS  REMAINING_PROGRESS_Region  6  (1) 


TIMER_STATIONS_Region 


NRC  DATA  Region  2(1) 
Gaze:  19.3% 


•we 


MAP  Region  4(1) 
Gaze:  36.2% 


LAST  TURN  RECRUITED_Region  5  (1) 
Gaze:  0.2% 


MISSED  GOAL  Region  3(1) 
Gaze:  0.0% 


Figure  4.  Make  Goal  Regions  of  Interest 
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C.  EQUIPMENT 

The  equipment  used  in  this  study  consists  of  two  desktop  computers  with 
Windows  7  operating  systems  and  two  eye -tracking  stereo  cameras.  The  cameras  detect 
reflected  infrared  light  from  a  subject’s  face  and  eyes  with  12  mm  lenses.  The  cameras 
are  calibrated  to  each  subject  to  find  reference  points  on  the  face  (outside  corners  of  the 
eyes  and  mouth),  which  enable  the  camera  to  monitor  the  subject’s  head  position, 
direction  and  duration  of  gaze,  as  well  as  workload  from  pupil  dilation.  This  information 
is  recorded  by  EyeWorks  record  on  desktop  computer  number  two  (which  also  runs  all  of 
the  other  tasks  in  the  study).  Desktop  number  one  is  used  only  to  run  FaceLAB  5.0.7  30. 

D.  PROCEDURES 

This  study  was  approved  by  the  NPS  IRB.  The  experiment  typically  took  less  than 
one  hour.  Upon  arrival  at  the  lab,  the  general  purpose  of  the  study  was  explained  to 
subjects;  the  experimenter  used  a  script  to  explain  the  tasks  and  task  requirements  to  each 
subject.  Subjects  were  not  told  whether  they  were  in  a  stress  or  control  group  or  what 
each  task  measured  until  the  debrief,  and  they  were  not  allowed  to  ask  questions  of  the 
experimenter  once  a  task  was  begun. 

First,  subjects  completed  the  IRB-approved  consent  form,  followed  by  the 
demographic  survey,  and  then  completed  the  tasks  in  the  following  order:  Stroop,  Go  No 
Go  and  AUT.  These  tasks  were  completed  for  the  larger  TRADOC  study  and  are  not  the 
focus  of  this  thesis.  The  eye-tracking  equipment  was  then  calibrated  to  the  subject.  Once 
calibration  was  successfully  accomplished,  the  subject  completed  the  Make  Goal  Task 
and  had  his/her  eye  tracking  recorded.  Next,  subjects  completed  the  Post  Make  Goal 
Accessions  Survey  and  the  Post  Task  Survey.  Finally,  if  the  subject  was  a  member  of  the 
stress  group  s/he  was  debriefed,  informing  him/her  that  s/he  was  a  member  of  a  stress 
group  and  that  the  stress  induction  experienced  was  part  of  the  experiment.  The 
experimenter  also  answered  any  questions  the  subject  may  have  had  about  the  study, 
procedures,  goals  and  potential  implications  and  uses  of  the  results  in  developing  future 
training  Army’s  2025  Vision  Plan. 
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III.  RESULTS 


A.  DATA  PREPARATION  AND  STATISTICAL  ANALYSIS 

1.  Data  Preparation 

The  raw  eye-tracking  data  was  in  the  form  of  two  spreadsheets.  One  spreadsheet 
contained  the  start  and  end  time  of  each  fixation,  as  well  as  the  fixation  duration  by  each 
ROI.  There  also  is  a  default  ROI  called  “empty,”  which  corresponds  either  to  when  the 
subject  blinked,  lost  eye  tracking  due  to  looking  away  from  the  screen,  or  looked  at  non- 
ROI  areas  of  the  screen.  The  “empty”  ROI  contributes  nothing  of  importance  to  this 
study  so  it  was  excluded  from  the  analysis.  The  second  spreadsheet  consisted  of  the 
workload  data.  The  eye-tracking  software  also  produces  a  video  of  each  subject’s  eye 
gaze  while  they  played  Make  Goal.  The  Make  Goal  Game  output  was  in  the  form  of  a 
comma  separated  value  (csv)  file  with  all  of  the  game  data,  including  the  number  of 
accessions  (the  actual  score)  and  regret  (number  of  recruits  missed).  Regret  was 
calculated  as  the  difference  between  the  total  number  of  recruits  that  could  possibly  have 
been  gained  by  selecting  the  optimal  location,  and  the  number  actually  gained  in  the 
location  selected. 

Due  to  the  nature  of  the  eye-tracking  equipment,  the  eye  tracker  would  begin 
before  the  subject  began  the  Make  Goal  game  and  end  after  the  subject  had  completed 
Make  Goal.  In  order  to  remedy  this  timing  issue  and  ensure  that  all  start  and  end  times 
were  accurate,  the  video  output  depicting  the  gaze  trace  of  each  subject  as  they  played  the 
Make  Goal  game  was  examined  for  the  exact  time  stamps  corresponding  to  the  beginning 
and  end  of  the  game.  This  data  was  compiled  in  a  spreadsheet  that  was  read  into  R  (R 
Core  Team  2015)  along  with  the  eye-tracking  data.  R  was  then  used  to  obtain  the 
averages  of  all  of  the  eye  tracking  and  Make  Goal  measures  (number  of  fixations, 
fixation  duration  overall,  fixation  duration  by  each  ROI,  workload,  number  of  fixation 
shifts,  shift  rate,  accessions,  regret,  %  optimal  selection,  and  time),  according  to  the 
definitions  listed  in  Chapter  II.  They  were  all  output  to  a  master  file,  which  was  used  for 
the  statistical  analysis.  All  of  the  measures  in  the  control  group  had  ( n  =  20)  observations, 
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while  the  stress  group  had  ( n  =  19)  observations,  as  one  of  the  stress  group  subjects  did 
not  yield  usable  eye-tracking  data. 

2.  Statistical  Analysis  Methods 

Two  sets  of  analyses  between  groups  were  carried  out:  (1)  comparing  the  stress 
and  control  groups,  and  (2)  comparing  high  and  low  performers.  Prior  to  conducting 
these  group  comparisons,  the  data  were  examined  in  order  to  determine  which  statistical 
test  would  be  appropriate.  As  much  of  the  data  turned  out  to  not  have  normal 
distributions,  the  following  non-parametric  tests  were  used  as  appropriate  according  to 
the  data:  the  Kruskal- Wallis  Test  and  2  Sample  Wilcoxon  Test.  Both  were  implemented 
in  R.  Note  that  the  boxplots  produced  in  this  thesis  were  created  using  the  ggplot  2 
package  in  R  (Wickham,  2009).  Data  sets  that  included  outliers  more  than  two  standard 
deviations  from  the  mean  were  analyzed  both  with  and  without  the  outliers  to  ensure  the 
results  were  not  being  affected  significantly  by  the  presence  of  outliers.  A  two-tailed 
alpha  level  of  0.05  was  used  for  all  tests.  Because  this  thesis  is  exploratory,  a  large 
number  of  comparisons  were  made.  Thus,  the  overall  probability  of  a  type  I  error  is 
greater  than  0.05. 

B.  COMPARISON  OF  STRESS  AND  CONTROL  GROUPS  ACCORDING  TO 

EYE  TRACKING  AND  MAKE  GOAL  MEASURES. 

1.  Comparison  of  Self-Reported  Stress  Levels  between  the  Stress  and 
Control  Groups 

In  the  post-experiment  surveys  administered  to  each  subject,  the  subjects  were 
asked  to  report  the  level  of  stress  they  experienced  during  the  Make  Goal  Game  in  terms 
of  how  stressful  they  found  the  game  in  general,  the  timer,  and  the  experimenter’s 
behavior  (induced  experimenter  stress).  Using  the  2  Sample  Wilcoxon  Test,  the  only 
significant  difference  found  was  between  the  rank  means  of  the  levels  of  induced 
experimenter  stress.  The  results  are  listed  in  Table  4  in  Appendix  B.  A  visual  comparison 
of  the  stress  and  control  groups  by  type  of  self-reported  stress  is  portrayed  in  Figure  5.  In 
addition  to  the  significant  difference  between  the  two  groups  with  regard  to  experimenter 
induced  stress,  there  appears  to  be  a  trend  for  the  stress  group  to  report  higher  stress 
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levels  in  all  three  stress  categories,  despite  several  outliers.  It  should  be  noted  that  in 
Figure  5,  the  black  lines  running  through  the  middle  of  the  boxplots  (but  seen  at  the  very 
bottom  of  the  induced  stress  plots  for  both  the  stress  and  control  groups)  represent  the 
median  stress  value.  The  boxes  themselves  measure  the  distance  from  the  lower  quartile 
values  to  the  upper  quartile  values,  as  measured  by  distance  from  the  median  value.  The 
“whiskers”  represent  values  1.5  times  the  interquartile  range  as  measured  from  the 
median.  The  individual  dots  represent  values  that  fall  outside  the  range  indicated  by  the 
“whiskers.”  Regarding  the  control  group  game  stress  plot,  the  dot  indicating  a  stress 
value  of  “4”  represents  two  subjects  reporting  a  stress  level  of  “4,”  while  the  dot  at  “7” 
indicates  that  one  individual  subject’s  response  was  a  “7.”  For  the  control  group  induced 
stress  plot,  the  dot  at  “2”  represents  the  fact  that  two  control  group  subjects  found  the 
experimenter’s  presence  during  the  experiment  to  be  stressful,  and  the  dot  at  “1” 
indicates  that  one  subject  reported  a  stress  level  of  “1.”  In  the  stress  group  game  stress 
plot  the  dot  at  “7”  represents  one  subject’s  response  of  a  stress  level  of  “7.” 
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Self-Reported  Stress  by  Experimental  Group 


Control 


Stress 


Experimental  Group 


Stress  Type 
E-El  game 

E^3  INDUCED 
TIMER 


Game  stress  refers  to  the  overall  game  stress,  induced  stress  refers  to  induced 
experimenter  stress,  and  timer  stress  refers  to  stress  caused  by  the  presence  of  a  timer  on 
the  game  screen.  The  specific  questions  used  to  elicit  these  self-reported  stress  levels  can 
be  found  in  Appendix  C. 


Figure  5.  Self-Reported  Stress  by  Experimental  Group  on  an  8-Point  Likert 

Scale 


2.  Comparison  of  Stress  and  Control  Groups  According  to  Eye-tracking 
Measures 

Using  the  Kruskal-Wallis  test,  there  were  no  significant  differences  found 
between  the  rank  means  of  the  stress  and  control  groups’  general  eye -tracking  measures, 
but  there  was  a  trend  for  the  control  group  to  have  a  greater  number  of  fixations  and 
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fixation  shifts.  This  trend  is  depicted  in  Figure  6.  A  tabular  comparison  is  given  in 
Table  5  in  Appendix  B. 


Fixation  Shift  Rate  &  Number  of  Fixations  by  Experimental  Group 
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Figure  6.  Comparison  of  Fixations  and  Fixation  Shifts  by  Experimental  Group 


The  control  group’s  trend  towards  more  fixations  is  depicted  graphically  in 
Figure  7  as  a  function  of  game  duration.  The  stress  group’s  fixations  over  time  are  also 
depicted  to  provide  a  visual  comparison.  There  appears  to  be  a  linear  relationship 
between  the  control  group’s  fixations  versus  game  duration,  implying  that  the  longer  the 
control  subjects  took  to  complete  the  game  the  more  fixations  they  demonstrated, 
whereas  the  stress  group  displays  less  of  a  linear  relationship  between  the  two  variables. 
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Figure  7.  Number  of  Fixations  by  Game  Duration 


There  was  only  one  significant  difference  in  the  mean  percentage  of  time  spent 
attending  to  each  ROI  (as  measured  by  the  Kruskal-Wallis  Test  using  rank  means).  The 
control  group  tended  to  look  at  the  Turns  Remaining/Progress  ROI  more  frequently  than 
did  the  stress  group.  A  tabular  summary  of  the  analysis  is  provided  in  Table  6,  Appendix 
B.  There  were  outliers  in  the  stress  group  for  the  percentage  of  time  spent  attending  to  the 
Missed  Goal  and  Timer/Stations  ROIs.  These  outliers  were  removed  and  the  statistical 
tests  were  repeated,  but  the  results  were  the  same  as  those  reported  in  Table  6.  Boxplots 
of  the  percentage  of  time  spent  fixating  on  each  ROI  are  shown  in  Figure  8. 
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Percentage  of  Fixation  Time  in  Each  ROI  by  Experimental  Group 
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Figure  8.  Stress  and  Control  Groups  Compared  According  to  Percentage  of 

Fixation  Time  per  ROI. 


Using  the  Kruskal- Wallis  test,  there  were  no  significant  differences  found 
between  the  stress  and  control  groups  in  terms  of  the  rank  means  of  duration  (in  seconds) 
that  subjects  in  each  group  looked  at  the  various  ROIs.  The  comparison  between  the 
stress  and  control  groups  is  depicted  graphically  in  Figure  9  and  in  tabular  form  in  Table 
7,  Appendix  B. 
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Fixation  Duration  (sec)  in  Each  ROI  by  Experimental  Group 
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Figure  9.  Stress  and  Control  Groups  Compared  According  to  Fixation  Duration 

in  Seconds  per  ROI. 

3.  Comparison  of  Stress  and  Control  Groups  According  to  Make  Goal 
Performance  Measures 

Figure  10  visually  summarizes  the  results  comparing  the  stress  and  control  groups 
according  to  the  Make  Goal  game  outcome  measures  (see  Table  2  in  Chapter  II  for  the 
definition  of  each  measure).  A  tabular  summary  can  be  found  in  Table  8,  Appendix  B;  no 
significant  differences  were  found  (using  the  Kruskal-Wallis  test)  between  the  rank 
means  of  the  stress  and  control  groups  in  terms  of  any  of  the  Make  Goal  measures.  An 
outlier  was  found  in  the  control  group  for  the  Accessions  variable  and  the  analysis  was 
redone,  with  no  change  in  results. 
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Comparison  of  Control  and  Stress  Groups  according  to  Make  Goal  Measures 
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Figure  10.  Experimental  Group  Comparison  According  to  Make  Goal  Measures 


C.  COMPARISON  OF  HIGH  AND  LOW  PERFORMERS  ACCORDING  TO 
EYE-TRACKING  AND  MAKE  GOAL  MEASURES 

The  next  group  comparison  made  was  between  the  high  and  low  Make  Goal 
performers.  High  performers  were  those  subjects  with  the  top  ten  accession  scores;  low 
performers  were  those  with  the  lowest  ten  accession  scores.  This  method  was  used 
because  it  yields  a  large  enough  sample  size  ( n  =  20)  to  be  of  use  statistically.  As 
demonstrated  in  Table  9,  Appendix  B,  the  Kruskal-Wallis  test  indicates  that  the  high 
performers  did  perform  significantly  better  in  terms  of  the  rank  mean  of  total  accessions 
and  percent  optimal  station  selection  than  did  the  low  performers,  despite  taking  similar 
amounts  of  time  to  complete  the  game.  The  next  step  was  to  determine  if  eye  tracking 
could  provide  insights  into  why  the  high  performers  did  so  much  better  in  terms  of  total 
accessions  gained  than  did  the  low  performers.  A  visual  comparison  between  the  high 

and  low  performing  groups  is  given  in  Figure  11. 
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Comparison  of  High  and  Low  Performers  according  to  Make  Goal  Measures 
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Figure  11.  High  and  Low  Performers  Compared  According  to  Make  Goal 

Measures 


1.  Comparison  of  High  and  Low  Performers  According  to  Eye-Tracking 
Measures 

Using  the  Kruskal-Wallis  test,  no  significant  differences  were  found  between  the 
top  and  bottom  ten  performers  according  the  rank  means  of  the  general  eye-tracking 
measures.  Graphically  the  groups  are  compared  in  Figures  12  and  13  according  to  these 
measures  (separate  graphs  were  used  in  order  to  compare  measures  on  similar  scales).  A 
tabular  summary  can  be  found  in  Table  10,  Appendix  B.  It  is  interesting  to  note  that  the 
top  ten  performers  included  five  members  of  the  stress  group  and  five  members  of  the 
control  group,  while  the  bottom  ten  performers  included  four  subjects  from  the  stress 
group  and  six  from  the  control  group — a  fairly  even  split. 
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Comparison  of  High  and  Low  Performers  according  to  Eye-Tracking  Measures 
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Figure  12.  Comparison  of  High  and  Low  Performers  According  to  Fixation 
Duration,  Cognitive  Workload  and  Fixation  Shift  Rate 
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Comparison  of  High  and  Low  Performers  according  to  Eye-Tracking  Measures 
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Figure  13.  Comparison  of  High  and  Low  Performers  According  to  Fixation  Shifts 

and  Number  of  Fixations 


Of  note  in  Figure  13  is  the  greater  variability  seen  in  the  high-performing  group’s 
fixations  and  fixation  shifts. 

Depicted  in  Figure  14  are  the  comparisons  between  the  high  and  low  performers 
according  to  the  percentage  of  time  spent  looking  at  each  ROI.  The  Kruskal- Wallis  test 
indicates  that  differences  between  the  two  groups  are  significant  for  the  rank  means  of 
percentage  of  time  spent  fixating  on  the  Missed  Goal  ROI  and  Turns  Remaining/Progress 
ROI.  The  high-performing  group  tended  to  look,  on  average,  more  frequently  at  the 
Missed  Goal  ROI  and  the  lower-performing  group  tended  to  look,  on  average,  more 
frequently  at  the  Turns  Remaining/Progress  ROI.  A  tabular  comparison  is  given  in 
Table  11,  Appendix  B. 
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Figure  14.  Comparison  of  High  and  Low  Performers  According  to  Percentage  of 

Fixation  Time  in  Each  ROI 
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Figure  15  depicts  the  comparison  of  the  top  and  bottom  ten  performers  according 
to  the  amount  of  time  in  seconds  that  they  spent  looking  at  each  ROI.  There  were  no 
significant  differences  found  in  the  rank  means  of  fixation  duration  using  the  Kruskal- 
Wallis  test.  Table  12  in  Appendix  B  depicts  these  comparisons  in  tabular  form. 


Fixation  Duration  (sec)  in  each  ROI  by  Performance  Group 
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Figure  15.  High  and  Low  Performers  Compared  According  to  Fixation  Duration 

in  Seconds  in  Each  ROI. 


Figure  16  depicts  results  of  the  analysis  of  the  self-reported  stress  of  the  high  and 
low  performers.  The  2  Sample  Wilcoxon  test  indicates  a  significant  difference  in  the  rank 
sums  of  timer  stress  between  the  high  and  low  performers,  with  the  high  performers 
reporting  a  significantly  higher  level  of  timer  stress  than  the  low  performers.  Note  that 
there  were  only  two  low-performing  subjects  who  reported  any  timer  stress,  as  indicated 
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by  the  dots  at  the  stress  values  of  “2”  and  “4.”  A  tabular  summary  of  the  comparison  can 
be  found  in  Table  13  in  Appendix  B. 
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Figure  16.  High  and  Low  Perfonnance  Groups  Compared  According  to  Self- 

Reported  Stress. 


2.  Summary  of  Results 

In  conclusion,  the  comparison  between  the  stress  and  control  groups  yielded  few 
significant  differences  in  rank  means  or  sums,  as  measured  by  the  Kruskal-Wallis  and  2 
Sample  Wilcoxon  tests,  respectively.  The  only  difference  found  was  that  the  control 
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group  spent  a  larger  percentage  of  time,  on  average,  looking  at  the  Turns  Remaining/ 
Progress  ROI  than  did  the  stress  group.  The  comparison  between  the  high  and  low 
performers  yielded  more  significant  differences,  however.  The  high  performers,  as  would 
be  expected,  had  a  higher  mean  number  of  accessions  and  a  higher  mean  incidence  of 
percent  optimal  station  selection.  High  performers  were  also  found,  on  average,  to  spend 
a  greater  percentage  of  time  looking  at  the  missed  goal  ROI  and  the  low  performers  spent 
more  time,  on  average,  looking  at  the  Turns  Remaining/Progress  ROI. 
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IV.  DISCUSSION 


A.  INTERPRETATION 

1.  Overall  Impact  of  Stress 

Interpreting  the  results  of  this  analysis  to  decipher  exactly  how  stress  impacted 
performance  is  complicated,  because  the  control  group  also  reported  experiencing  some 
level  of  stress  while  playing  Make  Goal,  whether  from  the  game  itself,  the  timer,  or 
having  the  experimenter  present  while  they  performed  the  task  (induced  experimenter 
stress),  a  point  discussed  below.  Additionally,  almost  no  significant  differences  were 
found  in  attention-allocation  patterns  and  workload  from  the  eye-tracking  data. 
Furthermore,  there  was  no  statistically  significant  difference  in  performance  between  the 
stress  and  control  groups  playing  Make  Goal,  as  depicted  in  Table  8  (Appendix  B,  p.  49). 

According  to  the  Yerkes-Dodson  stress  model  (Yerkes  &  Dodson,  1908), 
moderate  levels  of  stress  will  result  in  optimal  performance,  while  very  low  or  very  high 
levels  of  stress  result  in  degraded  performance.  It  is  possible  that  such  a  non-monotonic 
relationship  between  stress  and  performance  is  being  exemplified  in  these  results;  for 
example,  it  is  possible  that  on  average,  both  stress  and  control  subjects  experienced  low 
levels  of  stress,  which  resulted  in  similarly  modest  Make  Goal  performance  and  similar 
attention  allocation  patterns.  However,  the  experiment  would  require  a  greater  range  of 
stress  levels  in  order  to  correlate  specific  levels  of  stress  with  specific  points  along  the 
Yerkes-Dodson  curve,  as  shown  in  Figure  17.  This  experiment  only  contained  two  levels 
of  stress  (timer  only,  timer  plus  social  stressor)  which  cover  only  the  low  to  moderate 
stress  level  range  in  Figure  17.  Thus,  an  experiment  that  also  includes  a  no  stress  level 
and  a  high  stress  level  may  be  more  likely  to  detect  stress  effects  on  decision 
performance  and  attention  allocation  patterns. 

Interestingly,  the  high  performers  reported  the  timer  as  significantly  more 
stressful  than  the  low  performers.  It  may  be  the  case  that  this  pushed  them  to  moderate 
levels  of  stress,  aiding  their  performance.  In  contrast,  the  low  performers  appear  to  have 
experienced  low  levels  of  stress,  hurting  their  performance. 
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Source:  Pieterse,  Graydon,  n.d..  What’s  Stress  Got  to  Do  with  It?,  http://haleo.co.uk/ 
wordpress/whats-stress-got -to-do- with- it/ 

Figure  17.  Yerkes-Dodson  Inverted  U  Relationship  between  Performance  and 

Stress 


2.  Comparisons  between  the  Stress  Group  and  Control  Group  on  Eye 
Tracking  and  Game  Performance 

As  discussed  in  the  Results  section,  the  results  showed  that  there  was  only  one 
parameter  significantly  different  between  the  stress  and  control  groups,  with  the  control 
group  looking,  on  average,  more  frequently  at  the  Turns  Remaining/Progress  ROI.  There 
also  appeared  to  be  a  trend  for  the  stress  group  to  tend  to  have  more  fixations  than  the 
control  group.  It  is  unclear  why  exactly  the  control  group  fixated  more  on  the  Turns 
Remaining/Progress  ROI,  as  it  did  not  provide  any  important  decision  making 
information.  A  possible  explanation  is  that  it  is  natural  in  a  timed  process  to  want  to  keep 
track  of  time  left,  or  that  the  control  subjects  had  lost  interest  in  the  game  and  kept 
checking  to  see  when  it  would  be  over,  or  again,  they  may  have  been  frustrated  trying  to 
figure  the  game  out  and  were  checking  how  many  more  turns  they  had  to  do  so.  Control 
subjects  may  have  fixated  on  the  ROI  out  of  frustration  as  well;  the  progress  bar  displays 
an  accessions  goal  of  seven  hundred  twenty-five  recruits,  but  the  game  was  designed  so 
that  the  maximum  possible  number  of  accessions  was  six  hundred  five.  Subjects  may 
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have  been  stressed/panicked  looking  at  how  far  their  progress  always  remained  from  the 
displayed  goal. 

The  results  were  not  quite  what  was  predicted;  based  on  the  literature  and 
research  regarding  stress  and  its  effects  on  cognitive  processes  as  measured  by  eye 
tracking,  more  significant  differences  between  the  stress  and  control  groups  would  have 
been  expected.  The  stress  group  was  expected  to  perform  better  than  the  control  group, 
because  the  stress  was  assessed  to  be  moderate  and  in  the  range  likely  to  enhance 
performance  (according  to  the  Yerkes-Dodson  stress  model.)  However,  this  was  not  the 
case.  In  addition,  higher  workload  and  more  fixations  would  have  been  expected  for  the 
stress  group.  There  was  a  trend  for  the  stress  group  to  have  more  fixations,  which  makes 
sense,  even  if  it  was  not  statistically  significant.  It  is  possible  that  the  sample  sizes  in 
=20)  for  each  group  were  simply  not  large  enough  to  capture  these  expected  differences, 
or  that  the  sample  group  consisting  solely  of  military  officers  was  less  likely  to  be 
affected  by  the  experimental  stress  induction  than  an  equivalent  group  from  a  civilian 
university.  For  instance,  the  mild  to  moderate  stress  induced  for  the  stress  group  may  not 
have  been  enough  to  faze  these  specific  subjects.  Generally  speaking,  military  personnel 
have  at  some  point  in  their  careers  and  especially  in  basic  training  been  exposed  to 
stressors  such  as  being  yelled  at,  told  their  performance  is  miserable,  etc.  Thus,  the 
induced  experimental  stressor  may  have  been  something  they  were  easily  able  to  ignore. 

3.  Comparison  between  High  and  Low  Performers  on  Eye  tracking  and 
Game  Performance 

The  comparison  between  the  high-  and  low-performing  groups  showed  more 
significant  differences.  It  is  interesting  to  note  that  the  top  ten  scores  were  attained  by  the 
same  number  of  stress  group  subjects  as  control  group  subjects,  and  the  bottom  ten 
performers  included  six  members  of  the  control  group  and  four  members  of  the  stress 
group.  This  reinforces  the  finding  that  there  was  no  significant  difference  in  performance 
between  the  stress  and  control  groups.  As  would  be  expected,  the  high  performers  had 
more  accessions,  and  a  higher  incidence  of  optimal  station  selection.  The  eye-tracking 
information  helped  underscore  the  differences  in  attention  allocation  between  the  two 
groups.  The  differences  found  between  the  two  groups  were  statistically  significant  for 
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the  mean  percentage  of  total  fixations  on  the  Missed  Goal  ROI  and  the  mean  percentage 
of  total  fixations  on  the  Turns  Remaining/Progress  ROI,  with  the  high  performers  looking 
more  frequently  at  the  Missed  Goal  ROI  and  the  low  performers  looking  more  often  at 
the  Turns  Remaining/Progress  ROI.  This  is  in  keeping  with  what  was  expected:  the 
Missed  Goal  ROI  provided  the  most  useful  feedback  to  subjects  regarding  the  extent  to 
which  each  of  their  decisions  deviated  from  the  optimal  decision.  Turns  Remaining/ 
Progress  provided  no  useful  strategic  feedback  to  subjects,  so  it  is  not  surprising  that  the 
subjects  who  spent  more  time  looking  at  that  ROI  were  underperforming. 

B.  IMPLICATIONS 

This  was  an  exploratory  study  and  as  such  had  no  formal  hypotheses;  it  was 
hoped  that  the  cognitive  processes  of  better  decision  makers  could  be  quantified  by  the 
eye -tracking  data  collected,  and  that  the  effects  of  stress  on  cognitive  processes  could 
similarly  be  quantified  according  to  the  eye-tracking  data.  This  study  showed  that  higher 
performers  were  in  fact  better  able  to  figure  out  and  exploit  patterns  in  the  game,  by 
attending  to  the  most  salient  pieces  of  information.  The  higher  performers  also  reported 
experiencing  greater  stress  due  to  the  timer  in  the  game,  something  that  reinforces  the 
concept  of  the  Yerkes-Dodson  (1908)  model  of  a  non-monotonic  relationship  between 
stress  and  performance.  Overall,  this  study  has  implications  for  the  Army’s  goal  of 
quantifying  cognitive  agility  in  personnel.  Eye-tracking  information  is  a  measure  that  was 
shown  to  correlate  to  performance  and  provide  some  insight  into  what  information  high 
performers  chose  to  focus  on  versus  the  low  performers. 

C.  LESSONS  LEARNED 

Several  important  lessons  were  learned  from  this  study,  particularly  with  regard  to 
experimental  design.  Due  to  the  way  the  game  was  designed,  it  was  impossible  to  remove 
the  timer  feature  for  the  control  subjects  and  keep  it  for  the  stress  group.  Therefore,  the 
control  group  was  not  “stress-free.”  In  retrospect  it  is  not  that  surprising  that  the  stress 
and  control  groups  were  similar  in  terms  of  the  stress  the  subjects  experienced.  This 
pattern  made  interpreting  the  subjects’  performance  results  in  terms  of  stress  much  more 
complicated  because  both  groups  reported  finding  the  timer  stressful,  and  even  to  some 
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extent  having  an  experimenter  present,  even  without  stress  induction.  A  future 
experiment  could  benefit  from  this  lesson  and  design  an  experiment  in  which  the  control 
group  receives  a  more  stress-free  condition.  This  could  possibly  be  accomplished  by 
removing  the  time  limit  per  turn  (as  the  timer  was  reported  as  the  most  stressful  aspect  of 
the  game  by  the  control  group),  and  having  the  experimenter  leave  the  room.  This  way 
the  stress  results  would  not  be  confounded  by  having  similar  stressors  shared  by  both 
groups.  In  order  to  facilitate  replication  of  the  Yerkes-Dodson  curve  over  a  quantified 
range  of  stressors,  it  would  be  helpful  to  design  the  experiment  to  include  a  means  of 
objectively  quantifying  stress  on  a  continuous  scale,  perhaps  using  an  EEG,  heart  rate 
monitor,  or  cortisol  levels.  This  way  stress  levels  could  be  clearly  quantified  and  plotted 
against  performance,  which  would  be  very  informative. 

In  trying  to  interpret  the  results  of  the  eye-tracking  data  in  the  various  ROIs,  it 
was  found  that  the  control  group  and  low  performers  both  tended  to  look  at  the  Turns 
Remaining/Progress  ROI  more  frequently  than  the  other  ROIs.  As  was  discussed  above, 
there  are  several  possible  interpretations  for  this,  and  it  would  be  interesting  to  know 
what  the  individual  subject’s  motivations  and  logic  were  in  focusing  on  this  ROI.  A 
future  study  could  add  a  question  to  the  Post  Task  Survey  asking  subjects  to  explain  why 
they  looked  in  each  region  to  help  clarify  what  their  cognitive  process  was. 

D.  CONCLUSION 

The  ultimate  goal  of  the  overall  TRADOC  study  was  first,  to  identify  cognitively 
agile  military  personnel  and  second,  to  investigate  the  effects  of  stress  on  cognitive 
agility.  In  support  of  these  goals  the  aim  of  this  thesis  was  to  1)  determine  if  certain 
attention-allocation  patterns  are  associated  with  effective  cognitive-agility  performance, 
and  2)  to  investigate  the  effects  of  stress  on  cognitive  agility  as  measured  by  attention 
allocation.  The  study  was  a  success  in  that  the  eye-tracking  data  did,  in  fact,  provide 
insight  into  what  information  the  higher-performing  (more  cognitively  agile)  subjects 
attended  to,  revealing  common  attention-allocation  patterns  among  high  performers 
versus  low  performers.  In  terms  of  the  effects  of  stress  on  cognitive  agility,  the  precise 
effects  in  this  study  were  difficult  to  quantify  due  to  experimental  design  issues,  but 
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important  lessons  were  learned  that  will  hopefully  be  of  use  in  designing  and  executing 
future  studies. 

From  the  analysis  of  the  eye-tracking  data  collected  on  the  forty  subjects  who 
played  the  Make  Goal  game,  it  became  evident  that  there  was  only  one  significant 
difference  between  the  stress  and  control  groups.  This  was  an  unexpected  difference, 
which  involved  the  control  group  tending  to  fixate  more  frequently  on  an  ROI  containing 
non-essential  information  (Turns  Remaining/Progress  ROI).  High  performers  tended  to 
spend  a  greater  percentage  of  time  looking  at  the  missed  goal  ROI  (the  most  essential 
information)  while  the  low  performers  tended  to  spend  more  time  looking  at  the  Turns 
Remaining/Progress  ROI.  In  this  regard,  the  eye  tracking  did  provide  insight  into  what 
the  high  performers  chose  to  focus  on  versus  the  low  performers,  which  shows  promise 
for  future  cognitive-agility  studies. 

The  effects  of  stress  were  not  as  clear  from  the  analysis,  and  interestingly,  the 
initial  expectation  that  the  stress  group  would  perform  differently  was  not  borne  out  by 
the  statistical  analysis.  This  is  likely  the  case  because  the  control  group  was  not  entirely 
stress-free.  Based  on  this  finding  it  is  recommended  that  future  studies  use  a  greater 
number  of  stress  levels  and  incorporate  a  means  of  objectively  quantifying  stress  levels 
(i.e.,  via  skin  conductance,  EEG),  in  order  to  more  effective  apply  the  Yerkes-Dodson 
paradigm  to  the  analysis.  In  conclusion,  this  study  obtained  some  useful  results  in  terms 
of  using  eye-  tracking  data  to  determine  why  some  subjects  performed  well  while  others 
performed  poorly.  This  study  was  exploratory,  and  it  is  hoped  that  the  lessons  learned 
will  pave  the  way  for  future  study  in  this  area. 
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APPENDIX  A.  MAKE  GOAL  DATA  ATTRIBUTES  BY  TURNS 


Figures  18  to  23  depict  the  six  times  during  the  Make  Goal  game  that  the  optimal 
station  selection  and  most  important  attribute  changed,  as  referenced  in  Section  B,  part  1, 
describing  the  Make  Goal  task  (p.  10-11).  The  first  four  columns  labeled  “Laredo,” 
“McAllen,”  “Mercado”  and  “N.  Corpus”  are  all  recruiting  districts  in  which  subjects 
could  place  the  recruiter.  The  numbers  given  in  each  column  indicate  the  total  number  of 
recruits  that  could  be  gained  by  placing  the  recruiter  in  that  district  for  that  particular 
turn.  The  “Max  Goal”  column  indicates  the  highest  possible  number  of  recruits  that  could 
possibly  be  gained  on  that  turn  by  selecting  the  optimal  station.  The  “Time  per  Turn” 
column  indicates  the  number  of  seconds  subjects  had  on  that  particular  turn  to  place  the 
recruiter  in  a  given  district.  The  attribute  column  indicates  the  particular  data  attribute 
that  was  most  salient  (and  therefore  distinguished  the  optimal  station)  for  that  particular 
turn. 


Laredo 

McAllen 

Mercado 

N.  Corpus  Max  Goal 

Time  perTurn  (sec) 

Attribute 

Optimal  Station 

5 

3 

2 

0 

5 

100 

Diversity 

Laredo 

6 

4 

2 

0 

6 

100 

Diversity 

Laredo 

4 

3 

1 

2 

4 

100 

Diversity 

Laredo 

6 

0 

2 

1 

6 

100 

Diversity 

Laredo 

9 

7 

0 

4 

9 

100 

Diversity 

Laredo 

3 

4 

0 

1 

4 

100 

Diversity 

McAllen 

5 

4 

2 

3 

5 

90 

Diversity 

Laredo 

3 

3 

2 

3 

3 

90 

Diversity 

Laredo/McAllen/N. Corpus 

5 

4 

1 

2 

5 

90 

Diversity 

Laredo 

8 

0 

3 

2 

8 

90 

Diversity 

Laredo 

5 

6 

3 

2 

6 

90 

Diversity 

McAllen 

5 

4 

0 

0 

5 

90 

Diversity 

Laredo 

8 

6 

3 

3 

8 

80 

Diversity 

Laredo 

7 

4 

3 

3 

7 

80 

Diversity 

Laredo 

8 

6 

1 

1 

8 

80 

Diversity 

Laredo 

8 

5 

1 

1 

8 

80 

Diversity 

Laredo 

6 

4 

2 

0 

6 

80 

Diversity 

Laredo 

6 

6 

3 

1 

6 

80 

Diversity 

Laredo/McAllen 

10 

4 

1 

1 

10 

70 

Diversity 

Laredo 

4 

4 

1 

0 

4 

70 

Diversity 

Laredo/McAllen 

Figure  18.  Optimal  Attribute/Recruiting  Station  Selections  for  Moves  1-20  of 

Make  Goal 
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Figure  19.  Optimal  Attribute/Recruiting  Station  Selections  for  Moves  21-38  of 

Make  Goal 
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Figure  20.  Optimal  Attribute/Recruiting  Station  Selections  for  Moves  39-55  of 

Make  Goal 
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Figure  21.  Optimal  Attribute/Recruiting  Station  Selections  for  Moves  56-71  of 

Make  Goal 
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Figure  22.  Optimal  Attribute/Station  Selections  for  Moves  72-87  of  Make  Goal 
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Figure  23.  Optimal  Attribute/Recruiting  Station  Selections  for  Moves  88-100  of 

Make  Goal 
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APPENDIX  B.  DATA  ANALYSIS  SUMMARY  TABLES 


Table  4.  Subject  Self-Reported  Stress 


Stressor 

Control 

(mean,sd) 

95%  Cl 

Stress 

(mean,sd) 

95%  Cl 

Statistical  Test 

Result 

Game 

1.2(1.85) 

[0.33,  2.07] 

1.42(1.92) 

[0.49,  2.35] 

2  Sample  Wilcoxon 

W=  177 
p- value  =  0.7 1 

No  significant  difference 

Timer 

1.1  (1.33) 

[0.48,  1.72] 

1.95  (2.09) 

[0.94,  2.96] 

2  Sample  Wilcoxon 

W  =  151.5 
p-value=  0.26 

No  significant  difference 

Induced 

Stress 

0.25  (0.64) 

[0,  0.55] 

1.05  (1.27) 

[0.44,  1.66] 

2  Sample  Wilcoxon 

W  =  123 
p- value  =  0.02 

Significant  difference 

Table  5.  Comparison  of  Control  and  Stress  Groups  According  to  General  Eye- 

Tracking  Measures 


Measure 

Control 

(mean,sd) 

95%  Cl 

Stress 

(mean,sd) 

95%  Cl 

Statistical  Test 

Result 

Overall 

Fixation 

Duration 

0.45  (0.11) 
[0.39,0.50] 

0.46  (0.09) 
[0.42,0.50] 

Kruskal-Wallis 

Z2(l)  =  0.10 
p- value  =  0.18 

No  significant  difference 

Number  of 
Fixations 

1534  (689) 
[1211,1856] 

1224  (485) 
[990,1458] 

Kruskal-Wallis 

Z2  (1)  =  1.82 
p-Value=  0.18 

No  significant  difference 

Fixation 

Shifts 

1088  (350) 
903,1273] 

902  (340) 
[782,1066] 

Kruskal-Wallis 

z2(l)=1.82 
p- value  =  0.38 

No  significant  difference 

Fixation 

Shift  Rate 

0.73  (0.08) 
[0.69,0.77] 

0.75  (0.07) 
[0.71,0.78] 

Kruskal-Wallis 

z2(l)  =  0A3 
p-value=  0.7 1 

No  significant  difference 

Cognitive 

Workload 

0.51  (0.09) 
[0.47,0.55] 

0.48  (0.10) 
[0.43,0.53] 

Kruskal-Wallis 

z2(l)  =  038 
p- value  =  0.54 

No  significant  difference 
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Table  6.  Comparison  of  Control  and  Stress  Groups  According  to  Percentage  of 

Fixation  Time  in  Each  ROI 


Variable 
Name  ROI 

Control 

mean(sd) 

95%  Cl 

Stress 

mean(sd) 

95%  Cl 

Statistical 

Test 

Result 

Timer/ 

Stations 

36.36  (7.41) 
[32.89,39.83] 

35.95  (9.98) 
[31.14,40.76] 

Kruskal- Wallis 

(1)  =  0.08 
p-value  =  0.78 

No  significant  difference 

NRC  Data 

7.81  (3.13) 
[6.35,9.28] 

6.61  (5.26) 
[4.07,9.14] 

Kruskal-Wallis 

X1  (1)  =  1-20 
p-value  =  0.27 

No  significant  difference 

Missed  Goal 

11.64  (3.33) 
[10.09,13.20] 

13.55  (12.49) 
[7.53,19.57] 

Kruskal-Wallis 

^(1)  =  0.007 
p-value  =  0.93 

No  significant  difference 

Map 

40.35  (6.39) 
[37.36,43.34] 

40.77  (7.04) 
[37.38,44.16] 

Kruskal-Wallis 

x2(  1)  =  0 

p-value  =  1 

No  significant  difference 

Last  Turn 
Recruited 

2.23  (1.68) 
[1.44,2.23] 

2.40  (2.90) 
[1.00,2.40] 

Kruskal-Wallis 

X1  (1)  =  0.42 
p-value  =  0.52 

No  significant  difference 

Turns 

Remaining/ 

Progress 

1.61  (1.56) 
[0.88,2.34] 

0.73  (0.88) 
[0.30,1.15] 

Kruskal-Wallis 

z2  (11  =  4.56 
p-value  =  0.03 

Significant  difference 
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Table  7.  Comparison  of  Control  and  Stress  Groups  According  to  Fixation 

Duration  (in  Seconds)  in  Each  ROI 


Variable 
Name  ROI 

Control 

(mean,sd) 

95  %  Cl 

Stress 

(mean,sd) 

95%  Cl 

Statistical 

Test 

Result 

Timer/ 

Stations 

0.56  (0.19) 
[0.47,0.65] 

0.56  (0.13) 
[0.49,0.62] 

Kruskal- Wallis 

X20-)  =  0.003 
p-value  =  0.96 

No  significant  difference 

NRC  Data 

0.78  (0.32) 
[0.63,0.93] 

0.74  (0.37) 
[0.56,0.92] 

Kruskal-Wallis 

(1)  =  0.08 
p-value  =  0.78 

No  significant  difference 

Missed  Goal 

0.30  (0.08) 
[0.27,0.34] 

0.31  (0.11) 
[0.25,0.36] 

Kruskal-Wallis 

X~  (1)  =  0-08 
p-value  =  0.78 

No  significant  difference 

Map 

0.42  (0.10) 
[0.37,0.47] 

0.45  (0.10) 
[0.40,0.49] 

Kruskal-Wallis 

X2  (1)  =  0.71 
p-value  =  0.40 

No  significant  difference 

Last  Turn 
Recruited 

0.27(0.19) 

[0.18,0.36] 

0.25  (0.20) 
[0.15,0.34] 

Kruskal-Wallis 

X1  (1)  =  0-53 
p-value  =  0.47 

No  significant  difference 

Turns 

Remaining/ 

Progress 

0.45  (0.11) 
[0.39,0.50] 

0.46(0.09) 

[0.42,0.50] 

Kruskal-Wallis 

X1  (1)  =  0-78 
p-value  =  0.38 

No  significant  difference 

Table  8.  Comparison  of  Control  and  Stress  Groups  According  to  Make  Goal 

Measures 


Measure 

Control 

(mean,sd) 

95%  Cl 

Stress 

(mean,sd) 

95  %  Cl 

Statistical 

Test 

Result 

Accessions 

429  (56.26) 
[402.67,455.33] 

447.47  (41.43) 
[427.50,467.44] 

Kruskal-Wallis 

X2(l)=l.ll 

p-value  =  0.30 

No  significant 

Difference 

Regret 

176.0  (56.26) 
[149.67,  202.33] 

157.33  (41.43) 
[137.56,177.50] 

Kruskal-Wallis 

(1)=1.95 
p-value  =  0.16 

No  significant  difference 

%  Optimal 
Selection 

46.55  (11.60) 
[41.02,51.88] 

51.10(10.36) 

[46.11,56.10] 

Kruskal-Wallis 

X1  (1)  =  1-98 

p-value  =  0.16 

No  significant  difference 

Time 

777.55  (272.79) 
[649.89,905.22] 

798.49  (379.55) 
[615.55,981.42] 

Kruskal-Wallis 

X1  (1)  =  0-01 
p-value  =  0.93 

No  significant  difference 
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Table  9.  Comparison  of  High  and  Low  Performers  According  to  Make  Goal 

Measures 


Measure 

High  (mean,sd) 
95  %  Cl 

Low  (mean,sd) 
95  %  Cl 

Statistical 

Test 

Result 

Accessions 

490  (27.84) 
[470.09,  509.91] 

378.5  (50.30) 
[342.52,  414.48] 

Kruskal- Wallis 

;r(l)  =  14.32 
p-value  =  0.00 

Significant  difference 

Regret 

115  (27.84) 
[95.09,134.91] 

226.5  (50.29) 
[190.52,262.48] 

Kruskal- Wallis 

;r(D  =  14.32 
p-value  =  0.00 

Significant  difference 

%  Optimal 
Selection 

61  (8) 

[55.28,66.72] 

36.6  (9.36) 
[29.90,43.30] 

Kruskal-Wallis 

;r(l)  =  13.78 
p-value  =  0.00 

Significant  difference 

Time 

627.13  (310.06) 
[405.32,848.93] 

828.96  (373.00) 
[562.14,1095.79] 

Kruskal-Wallis 

z2(l)  =  1.85 
p-value  =  0.17 

No  significant  difference 

Table  10.  Comparison  of  High  and  Low  Performers  According  to  General  Eye- 

Tracking  Measures 


Measure 

High  (mean,sd) 
95  %  Cl 

Low  (mean,sd) 
95  %  Cl 

Statistical  Test 

Result 

Overall 

Fixation 

Duration 

0.43  (0.09) 
[0.37,0.49] 

0.44(0.13) 

[0.35,0.54] 

Kruskal-Wallis 

;T  0  =  0.09 
p-value  =  0.76 

No  significant  difference 

Number  of 
Fixations 

1274.7  (680.66) 
[787.79,1761.61] 

1513.8  (756.63) 
[972.54,2055.06] 

Kruskal-Wallis 

;r  (1)  =  1-29 
p-value=  0.26 

No  significant  difference 

Fixation 

Shifts 

944.6  (445.77) 
[625.72,1263.48] 

1063.3  (401.35) 
[776.19,1350.41] 

Kruskal-Wallis 

z2(l)  =  1.29 
p-value  =  0.26 

No  significant  difference 

Fixation 

Shift  Rate 

0.76  (0.06) 
[0.72,0.80] 

0.73  (0.08) 
[0.67,0.79] 

Kruskal-Wallis 

(1)  =  0.82 
p-value=  0.36 

No  significant  difference 

Cognitive 

Workload 

0.49  (0.09) 
[0.42,0.56] 

0.47  (0.09) 
[0.41,0.54] 

Kruskal-Wallis 

^2  (1)  =  0.37 
p-value  =  0.54 

No  significant  difference 

48 


Table  11.  Comparison  of  High  and  Low  Performers  According  to  Percentage  of 

Fixation  Time  in  Each  ROI 


Variable 
Name  ROI 

High 

mean(sd) 

95%  Cl 

Low 

mean(sd) 

95%  Cl 

Statistical  Test 

Result 

Timer/ 

Stations 

36.86  (6.52) 
[32.19,41.52] 

38.36  (7.06) 
[33.30,43.41] 

Kruskal-Wallis 

(1)  =  0.57 
p-value  =  0.45 

No  significant  difference 

NRC  Data 

5.72  (3.80) 
[3.00,8.44] 

7.16  (4.37) 
[4.04,10.28] 

Kruskal-Wallis 

X~  (1)  =  0-57 
p-value  =  0.45 

No  significant  difference 

Missed  Goal 

13.87  (3.25) 
[11.54,  16.19] 

8.98  (3.59) 
[6.41,11.55] 

Kruskal-Wallis 

X1  (1)  =  6-61 

p-value  =  0.01 

Significant  difference 

Map 

40.95  (5.73) 
[36.85,45.05] 

40.52  (4.94) 
[36.99,44.06] 

Kruskal-Wallis 

(11  =  0.37 
p-value  =  0.54 

No  significant  difference 

Last  Turn 
Recruited 

2.30  (2.32) 
[0.64,3.96] 

3.45(2.96) 

[1.33,5.57] 

Kruskal-Wallis 

z2  (11  =  0.97 
p-value  =  0.33 

No  significant  difference 

Turns 

Remaining/ 

Progress 

0.31  (0.31) 
[0.08,0.53] 

1.54(1.31) 
[0.60,  2.48] 

Kruskal-Wallis 

X~  (1)  =6.23 
p-value  =  0.01 

Significant  difference 
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Table  12.  Comparison  of  High  and  Low  Performers  According  to  Fixation 

Duration  (in  Seconds)  in  Each  ROI 


Variable 
Name  ROI 

High 

(mean,sd) 

95  %  Cl 

Low 

(mean.sd) 
95%  Cl 

Statistical  Test 

Result 

Timer/ 

Stations 

0.49  (0.13) 
[0.40.0.59] 

0.59  (0.19) 
[0.46,0.73] 

Kruskal-Wallis 

(1)  =  0.97 
p-value  =  0.32 

No  significant  difference 

NRC  Data 

0.73  (0.29) 
[0.53,0.93] 

0.77  (0.29) 
[0.56,0.97] 

Kruskal-Wallis 

X~  (1)  =  0-28 

p-value  =  0.60 

No  significant  difference 

Missed  Goal 

0.32  (0.07) 
[0.27,0.37] 

0.26  (0.11) 
[0.18,0.33] 

Kruskal-Wallis 

X1  (1)  =  0-08 
p-value  =  0.78 

No  significant  difference 

Map 

0.42  (0.10) 
[0.37,0.47] 

0.45  (0.10) 
[0.40,0.49] 

Kruskal-Wallis 

(11=  1.65 
p-value  =  0.20 

No  significant  difference 

Last  Turn 
Recruited 

0.26  (0.12) 
[0.35,0.52] 

0.42  (0.13) 
[0.17,0.35] 

Kruskal-Wallis 

;r  (1)  =  0-01 

p-value  =  0.47 

No  significant  difference 

Turns 

Remaining/ 

Progress 

0.45  (0.11) 
[0.39,0.50] 

0.46(0.09) 

[0.42,0.50] 

Kruskal-Wallis 

z2  (11  =  0.78 
p-value  =  0.94 

No  significant  difference 

Table  13.  Exploratory  Analyses:  Comparison  of  Self-Reported  Stress  Levels 

between  High  and  Low  Performers 


Stressor 

High 

(mean,sd) 

95%  Cl 

Low 

(mean,sd) 

95%  Cl 

Statistical  Test 

Result 

Game 

0.8  (0.79) 
[0.24,1.36] 

1.40  (2.37) 

[0,  3.09] 

2  Sample  Wilcoxon 

W=  53 

p-value  =  0.84 

No  significant  difference 

Timer 

1.40(1.07) 

[0.63,2.17] 

0.60(1.35) 

[0,  1.57] 

2  Sample  Wilcoxon 

W  =  78.5 
p-Value=  0.02 

Significant  difference 

Induced 

Stress 

0.40  (0.70) 

[0,  0.90] 

0.90(1.20) 

[0.04,  1.76] 

2  Sample  Wilcoxon 

W  =  40.5 
p-value  =  0.42 

No  significant  difference 
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APPENDIX  C.  POST  TASK  SURVEY 


Figure  24  depicts  the  two  pages  of  the  Post  Task  Survey  administered  to  subjects 
immediately  after  they  completed  the  Make  Goal  task. 


Postgame  Survey 


Subject  ID: _ _  Date: 

1 )  What  was  your  general  strategy  for  assigning  recruiters? 


2)  Did  your  strategy  change  during  the  game?  If  so,  how? 


3)  Which  items  of  information  did  you  find  most  useful? 


4)  How  challenging  did  you  find  the  game? 

01  2345678 

Not  at  all  Very 

challenging  challenging 

b)  If  you  circled  a  I  or  higher,  which  aspects  of  the  game  were  challenging? 


5)  How  stressful  did  you  find  the  game? 

012345678 
Not  at  all  Very 

stressful  stressful 


b)  If  you  circled  a  1  or  higher,  which  aspects  of  the  game  were  stressful? 


NPS  I RB 

APPROVED  Ap«  ?2  2016 

EXPIRED  10  m 


Figure  24.  Make  Goal  Post  Task  Survey  (Continued  on  Next  Page) 
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6)  Towards  the  end  of  the  game,  how  stressful  did  you  find  the  timer? 


01  2345678 

Not  at  all  Very 

stressful  stressful 

7)  How  stressful  did  you  find  the  experimenter’s  behavior  while  you  played  the 
game? 

01  2345678 

Not  at  all  Very 

stressful  stressful 


b)  If  you  circled  a  1  or  higher,  which  aspects  of  their  behavior  were  stressful? 


8)  Any  other  comments? 


NPS  I RB 

APPROVED •  ? 2  2016 
EXPIRED  APR  10  2017 


Figure  24.  (Continued  from  Previous  Page) 
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